Gitlab CI/CD example with Laravel, Kubernetes and Helm

03 Jan 2022DevOps Gitlab

Here is a full configuration file for Continuous Integration / Continuous Delivery (CI/CD) pipeline with Gitlab.

This is a three part article, meaning if you want explanations, just go down.

Entire Configuration file
Runner installation explanations
Explanation about the configuration file

Here are some technical consideration used in the post

Gitlab
2 Kubernetes clusters for production and staging
2 gitlab-runner shell and docker
Laravel
Helm

UPDATE 31/03/2022: more recent blog post but with less comments can be view here

Entire Configuration file

# .gitlab-ci.yaml
variables:
  IMAGE: eu.gcr.io/xxx/xxx
  TAG: "${CI_COMMIT_SHORT_SHA}"
  CONNECT_K8S_PREPROD_CMD: gcloud container clusters get-credentials xxx --zone europe-west1-b --project xxx
  CONNECT_K8S_PROD_CMD: gcloud container clusters get-credentials xxx --region europe-west1 --project xxx
  GIT_DEPTH: 1

stages:
  - build
  - test
  - maintenance_down
  - migration
  - delivery

build:
  stage: build
  tags:
    - gcp-shell
  except:
    - tags
  script:
    - docker build -t ${IMAGE}:${TAG} .
    - gcloud auth activate-service-account gitlab-push-container@xxx.iam.gserviceaccount.com --key-file=/gitlab.json
    - gcloud auth configure-docker
    - docker push ${IMAGE}:${TAG}

test:
  stage: test
  image: ${IMAGE}:${TAG}
  services:
    - mysql:latest
  variables:
    MYSQL_DATABASE: nci_authentication
    MYSQL_ROOT_PASSWORD: root
  tags:
    - gcp-docker
  except:
    - tags
  before_script:
    - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
    - cd /var/www/html
    - composer install
    - apk add gettext # envsubst
    - envsubst < /var/www/html/devops/.env.test > /var/www/html/.env
    - php artisan key:generate
    - php artisan migrate
  script:
    - ./vendor/phpunit/phpunit/phpunit

maintenance_down:
  stage: maintenance_down
  variables:
    CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
    - if: $CI_COMMIT_BRANCH == "develop"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
    - when: never
  tags:
    - gcp-shell
  script:
    - ${CONNECT_CLUSTER}
    - kubectl get po -l app.kubernetes.io/name=xnet -o name | xargs -I{} kubectl exec {} -c xnet-backend -- php artisan down

migration:
  stage: migration
  variables:
    CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
    ENV_FILE: /var/www/html/devops/.env.stage
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
        ENV_FILE: /var/www/html/devops/.env.prod
    - if: $CI_COMMIT_BRANCH == "develop"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
        ENV_FILE: /var/www/html/devops/.env.stage
    - when: never
  image: ${IMAGE}:${TAG}
  tags:
    - gcp-docker
  before_script:
    - apk add gettext # envsubst
    - envsubst < ${ENV_FILE} > /var/www/html/.env
    - cd /var/www/html
    - php artisan key:generate
  script:
    - cd /var/www/html
    - php artisan migrate

delivery:
  stage: delivery
  variables:
    CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
    VALUES_FILE: devops/helm/stage.yaml
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
        VALUES_FILE: devops/helm/prod.yaml
    - if: $CI_COMMIT_BRANCH == "develop"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
        VALUES_FILE: devops/helm/stage.yaml
    - when: never
  tags:
    - gcp-shell
  script:
    - ${CONNECT_CLUSTER}
    - envsubst < ${VALUES_FILE} > devops/helm/dist.yaml
    - helm upgrade -f devops/helm/dist.yaml xnet devops/helm/

About the runners and installation

Before even talk about the CI/CD himself, let's see how runners was setup. I'm sure I could come up with several ways to handle this.

The installation of the gitlab-runner is done with docker.
Two runners was registred, one with the docker mode, the other with shell mode.

On a fresh compute instance, docker was installed

curl https://baltocdn.com/helm/signing.asc | apt-key add -
apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list

Few more binary was install, it will be available on the shell runner.

Kubectl and HELM

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

apt-get update
apt-get install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubectl

We need gcloud as the container registry is on Google Cloud Platform.

apt-get install apt-transport-https ca-certificates gnupg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
apt-get update && apt-get install google-cloud-sdk
cat <<EOT >> /gitlab.json
{
  "type": "service_account",
  "project_id": "xxx",
  "private_key_id": "2xxxd",
  "private_key": "-----BEGIN PRIVATE KEY-----\xxx\n-----END PRIVATE KEY-----\n",
  "client_email": "xxx@xxx.iam.gserviceaccount.com",
  "client_id": "1xxx2",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxiam.gserviceaccount.com"
}
EOT
gcloud auth activate-service-account xxx@xxx.iam.gserviceaccount.com --key-file=/gitlab.json
gcloud auth configure-docker

This will usefull to manipulate some files like envsubst < ${VALUES_FILE} > devops/helm/dist.yaml.

apt-get install gettext-base

Interact with the runner

You'll might need interact with the gitlab-runner container. You can go inside the container and executes commandes like this.

docker exec -it gitlab-runner /bin/bash
# Now you are inside the container
cat /etc/gitlab-runner/config.toml
gitlab-runner register
gitlab-runner reload
gitlab-runner restart

Or send commande from outside the container like this, it will output the result.

docker exec gitlab-runner gitlab-runner verify

Configuration file explanation

variables

IMAGE: The docker registry image
TAG: This is the gitlab variable for git sha1 commit
CONNECT_K8S_* command line to connect to the production or staging Kubernetes.
GIT_DEPTH: we take only the last commit history for the build.

Build

build:
  stage: build
  tags:
    - gcp-shell
  except:
    - tags
  script:
    - docker build -t ${IMAGE}:${TAG} .
    - gcloud auth activate-service-account gitlab-push-container@xxx.iam.gserviceaccount.com --key-file=/gitlab.json
    - gcloud auth configure-docker
    - docker push ${IMAGE}:${TAG}

It will be run on the shell runner.
The gitlab.json is the authentication key for service account. It was upload on the runner once for all.
Build and push to the Google Cloud Registry. The key needs write access to the registry.

test

test:
  stage: test
  image: ${IMAGE}:${TAG}
  services:
    - mysql:latest
  variables:
    MYSQL_DATABASE: xnet
    MYSQL_ROOT_PASSWORD: root
  tags:
    - gcp-docker
  except:
    - tags
  before_script:
    - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
    - cd /var/www/html
    - composer install # on a besoin des dépendences de dev (contrairement à la prod)
    - mv /var/www/html/devops/.env.test /var/www/html/.env
    - php artisan key:generate
    - php artisan migrate
  script:
    - ./vendor/phpunit/phpunit/phpunit

It will be run on the docker runner. It will run a container on the exact image previously build, mount a MySQL as a service and run all tests inside the container.
Dependencies like composer must be installed.

# .env.test
APP_NAME=Laravel
APP_ENV=local
APP_KEY=

DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=xnet
DB_USERNAME=root
DB_PASSWORD=root

This .env.test will contains what is needed for running tests.

Maintenance down

maintenance_down:
  stage: maintenance_down
  variables:
    CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
    - if: $CI_COMMIT_BRANCH == "develop"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
    - when: never
  tags:
    - gcp-shell
  script:
    - ${CONNECT_CLUSTER}
    - kubectl get po -l app.kubernetes.io/name=xnet -o name | xargs -I{} kubectl exec {} -c xnet-backend -- php artisan down

php artisan down will be execute on every containers of the cluster.

Migration

migration:
  stage: migration
  variables:
    CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
    ENV_FILE: /var/www/html/devops/.env.stage
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
        ENV_FILE: /var/www/html/devops/.env.prod
    - if: $CI_COMMIT_BRANCH == "develop"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
        ENV_FILE: /var/www/html/devops/.env.stage
    - when: never
  image: ${IMAGE}:${TAG}
  tags:
    - gcp-docker
  before_script:
    - apk add gettext # envsubst
    - envsubst < ${ENV_FILE} > /var/www/html/.env
    - cd /var/www/html
    - php artisan key:generate
  script:
    - cd /var/www/html
    - php artisan migrate

$DB_PASSWORD_PREPROD is a variable set inside gitlab with Settings > CI/CD > Variables. envsubst will replace the value from a file to an output.
Shell runner

APP_NAME=Laravel
APP_ENV=local
APP_KEY=

DB_CONNECTION=mysql
DB_HOST=10.10.10.10
DB_PORT=3306
DB_DATABASE=xnet
DB_USERNAME=foo
DB_PASSWORD=$DB_PASSWORD_PREPROD

Delivery

delivery:
  stage: delivery
  variables:
    CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
    VALUES_FILE: devops/helm/stage.yaml
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
        VALUES_FILE: devops/helm/prod.yaml
    - if: $CI_COMMIT_BRANCH == "develop"
      when: on_success
      variables:
        CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
        VALUES_FILE: devops/helm/stage.yaml
    - when: never
  tags:
    - gcp-shell
  script:
    - ${CONNECT_CLUSTER}
    - envsubst < ${VALUES_FILE} > devops/helm/dist.yaml
    - helm upgrade -f devops/helm/dist.yaml xnet devops/helm/

Shell runner
helm values.yaml contains env that will be included, like the following file. envsubst will replace with the gitlab value.

env:
  LOG_CHANNEL: stderr
  DB_PASSWORD: $DB_PASSWORD_PREPROD

Summary

This configuration allow a full build and deployment from a Laravel build to a production Kubernetes cluster.
There is other way to approch the problem, here is the one we will have in production.

Next: X-Forwarded-For with Istio, Kubernetes and GCP load balancer
Previous: I'm not DevOps, I'm a DevCloud guy

Gitlab CI/CD example with Laravel, Kubernetes and Helm

Entire Configuration file #

About the runners and installation #

Interact with the runner #

Configuration file explanation #

variables #

Build #

test #

Maintenance down #

Migration #

Delivery #

Summary #